feat(vortex-array): execute primitive interleave arrays by HarukiMoriarty · Pull Request #9291 · vortex-data/vortex

HarukiMoriarty · 2026-08-07T20:04:04Z

Summary

execute Interleave arrays whose values use primitive dtypes
gather from canonical primitive buffers while preserving constant value sources
share selector length and bounds validation with the Boolean kernel
cover nullable primitive columns, constants, null constants, and invalid selectors

Motivation

Interleave previously only had an execution kernel for Boolean values. This adds the corresponding primitive kernel as a general vortex-array capability. Spatial MakeLine uses this support in the separate stacked PR #9201.

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

codspeed-hq · 2026-08-07T20:18:21Z

Merging this PR will regress 1 benchmark

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 1936 untouched benchmarks
🆕 12 new benchmarks
⏩ 89 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`cold_misaligned[(64, 256)]`	4.4 ms	5.3 ms	-17.42%
⚡	Simulation	`decode_varbin[(1000, 4)]`	99.3 µs	60.5 µs	+64.15%
⚡	Simulation	`slice_primitive_tight_loop[10000]`	502.7 µs	438.2 µs	+14.74%
⚡	Simulation	`slice_dict_tight_loop[10000]`	837.8 µs	738.6 µs	+13.44%
🆕	Simulation	`primitive_mixed_constants[random/n2/nonnull]`	N/A	420.5 µs	N/A
🆕	Simulation	`primitive_mixed_constants[random/n2/null]`	N/A	429.1 µs	N/A
🆕	Simulation	`primitive_mixed_constants[random/n64/nonnull]`	N/A	650.7 µs	N/A
🆕	Simulation	`primitive_mixed_constants[random/n64/null]`	N/A	714.6 µs	N/A
🆕	Simulation	`primitive_mixed_constants[round_robin/n2/nonnull]`	N/A	355.9 µs	N/A
🆕	Simulation	`primitive_mixed_constants[round_robin/n2/null]`	N/A	357.2 µs	N/A
🆕	Simulation	`primitive[random/n2/nonnull]`	N/A	538.1 µs	N/A
🆕	Simulation	`primitive[random/n2/null]`	N/A	545.2 µs	N/A
🆕	Simulation	`primitive[random/n64/nonnull]`	N/A	869.9 µs	N/A
🆕	Simulation	`primitive[random/n64/null]`	N/A	908.7 µs	N/A
🆕	Simulation	`primitive[round_robin/n2/nonnull]`	N/A	376.6 µs	N/A
🆕	Simulation	`primitive[round_robin/n2/null]`	N/A	383.9 µs	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing nemo/interleave-primitive-execution (a0ad6c9) with develop (e77f520)²}

89 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on develop (b4b8f54) during the generation of this report, so e77f520 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

joseph-isaacs · 2026-08-10T13:54:08Z

+    let len = validate_selectors(values.len(), |branch| values[branch].len(), branches, rows)?;
+    let mut output = BufferMut::with_capacity(len);
+    for i in 0..len {
+        output.push(values[branches[i].as_()].value(rows[i].as_()));


why do you validate once and then used checked accesses?

You likely want to zip with idx with the output and avoid the bounds check on the output

joseph-isaacs · 2026-08-10T13:55:51Z

    if array.value(0).dtype().is_boolean() {
        bool::execute(array, ctx)
+    } else if array.value(0).dtype().is_primitive() {
+        primitive::execute(array, ctx)
    } else {
        let value_dtype = array.value(0).dtype().clone();
        vortex_panic!(
-            "interleave execution is only implemented for boolean values; value dtype {} is not \
-             yet supported",
+            "interleave execution is not implemented for value dtype {}",
            value_dtype


I know this is my bad code but the return dtype can we used not the value(0) type

joseph-isaacs · 2026-08-10T13:56:56Z

+        if array.value(i).as_opt::<Constant>().is_none() {
+            array = require_child!(array, array.value(i), i + 2 => Primitive);
+        }


can use Columnar here

joseph-isaacs · 2026-08-10T13:58:09Z

+    _ctx: &mut ExecutionCtx,
+) -> VortexResult<ExecutionResult> {
+    let num_values = array.num_values();
+    array = require_child!(array, array.array_indices(), 0 => Primitive);


I think this could be bool if thre are two values. I cannot remember if we kept that?

Technically yes for exactly two values. The current Interleave implementation explicitly rejects Boolean array_indices, including the test from the original encoding PR, so I assumed that restriction was intentional. I’ve left it unchanged, but can add the two-value Boolean special case if that is now desired.

vortex/vortex-array/src/arrays/interleave/mod.rs

Lines 36 to 40 in fb53f39

//! ## Selector types

//!

//! `array_indices` encodes the value array per row as a non-nullable **unsigned integer**

//! (`array_indices[i]` is the index into `values`). `row_indices` is likewise a non-nullable

//! **unsigned integer** naming the position within the selected value array.

vortex/vortex-array/src/arrays/interleave/mod.rs

Lines 162 to 179 in fb53f39

// `row_indices` names a position within the selected value.

for (name, selector) in [

("array_indices", array_indices),

("row_indices", row_indices),

] {

match selector.dtype() {

DType::Primitive(ptype, nullability) if ptype.is_unsigned_int() => {

vortex_ensure!(

!nullability.is_nullable(),

"interleave {name} must be non-nullable, got {}",

selector.dtype()

);

}

other => vortex_bail!(

"interleave {name} must be a non-nullable unsigned integer, got {other}"

),

}

}

What do you think we should do here

joseph-isaacs · 2026-08-10T13:59:07Z

+enum PrimitiveValues<T> {
+    Buffer(Buffer<T>),
+    Constant { value: T, len: usize },
+}


Don't we have this code everywhere? Also you don't need the len

Removed the len. Existing types are either type-erased or operation-specific; I think this keeps the hot loop typed and local minimum.

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

joseph-isaacs · 2026-08-10T15:16:45Z

+        let row = (*row).as_();
+        // SAFETY: the caller guarantees that the selected branch and row are in bounds for
+        // `values` and the selected physical value buffer.
+        slot.write(unsafe { values.get_unchecked(branch).value_unchecked(row) });


Why do you want to do this unchecked vs just using the checked one?

joseph-isaacs · 2026-08-10T15:18:14Z

+            Self::Buffer(values) => *unsafe { values.get_unchecked(index) },
+            Self::Constant(value) => *value,


we could convert this from a branch into a offset load for the constant?

values[index & ty] where ty = usize::MAX if buffer and zero otherwise.

I am not sure what is faster

This one is actually faster.

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

## Rationale Vortex native geometry arrays should support constructing two-point LineStrings directly from paired Point columns without an Arrow or WKB round trip. ## What changes are included? - Add vortex.st.make_line for paired native Point arrays. - Promote mixed XY, XYZ, XYM, and XYZM inputs, filling absent ordinates with zero. - Propagate CRS metadata, constants, and endpoint nulls. - Build native LineString storage directly and add focused tests and CodSpeed benchmarks. ## Stack #9291 (primitive Interleave execution) has merged into develop, so this PR is now standalone: rebased onto develop, it contains only the spatial function. ST_Length was split into #9290 so each scalar function can also be reviewed independently. Signed-off-by: Nemo Yu <zyu379@wisc.edu>

feat(vortex-array): execute primitive interleave arrays

0b5c2ac

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

HarukiMoriarty requested a review from joseph-isaacs August 7, 2026 20:07

HarukiMoriarty added the changelog/feature A new feature label Aug 7, 2026

HarukiMoriarty mentioned this pull request Aug 7, 2026

feat(vortex-spatial): add make-line scalar function #9201

Merged

refactor(vortex-array): clarify interleave primitive values

6ad96f8

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

joseph-isaacs reviewed Aug 10, 2026

View reviewed changes

HarukiMoriarty added 2 commits August 10, 2026 10:04

perf(vortex-array): avoid redundant interleave bounds checks

ca65600

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

refactor(vortex-array): streamline primitive interleave gather

eadb2df

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

HarukiMoriarty requested a review from joseph-isaacs August 10, 2026 14:31

joseph-isaacs reviewed Aug 10, 2026

View reviewed changes

refactor(vortex-array): simplify interleave selector validation

a0ad6c9

Signed-off-by: Nemo Yu <zyu379@wisc.edu>

HarukiMoriarty requested a review from joseph-isaacs August 10, 2026 16:32

joseph-isaacs approved these changes Aug 10, 2026

View reviewed changes

HarukiMoriarty merged commit 9157d6c into develop Aug 10, 2026
82 of 83 checks passed

HarukiMoriarty deleted the nemo/interleave-primitive-execution branch August 10, 2026 16:54

	//! ## Selector types
	//!
	//! `array_indices` encodes the value array per row as a non-nullable unsigned integer
	//! (`array_indices[i]` is the index into `values`). `row_indices` is likewise a non-nullable
	//! unsigned integer naming the position within the selected value array.

	// `row_indices` names a position within the selected value.
	for (name, selector) in [
	("array_indices", array_indices),
	("row_indices", row_indices),
	] {
	match selector.dtype() {
	DType::Primitive(ptype, nullability) if ptype.is_unsigned_int() => {
	vortex_ensure!(
	!nullability.is_nullable(),
	"interleave {name} must be non-nullable, got {}",
	selector.dtype()
	);
	}
	other => vortex_bail!(
	"interleave {name} must be a non-nullable unsigned integer, got {other}"
	),
	}
	}

		Self::Buffer(values) => *unsafe { values.get_unchecked(index) },
		Self::Constant(value) => *value,

Uh oh!

Conversation

HarukiMoriarty commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Uh oh!

codspeed-hq Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 1 benchmark

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HarukiMoriarty commented Aug 7, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 7, 2026 •

edited

Loading